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Executive Summary 


This rapid evidence assessment aimed to examine the potential impact of school closures on the 
attainment gap, based on a systematic search of existing literature. Eleven studies were identified that 
provided quantitative evidence about the impact of school closures on attainment gaps. A subset of 
nine studies provided comparable estimates that could be synthesised. 


Although the search included school closures with a range of causes (including due to adverse weather, 
hurricanes, teacher industrial action and pandemics) the final group of robust estimates all came from 
studies of summer holidays for primary-aged children. 


Key findings and implications 


1. School closures are likely to reverse progress made to narrow the gap in the last decade 


The projections suggest that school closures will widen the attainment gap between disadvantaged 
children and their peers, likely reversing progress made to narrow the gap since 2011. The median 
estimate indicates that the gap would widen by 36%. 


However, the estimated rate of gap widening varied substantially between studies, meaning that there 
is a high level of uncertainty around this average. Plausible “good” and “bad” estimates range from the 
gap widening from 11% to 75%. 


2. Supporting effective remote learning will mitigate the extent to which the gap widens 


Pupils can learn through remote teaching. However, ensuring the elements of effective teaching are 
present — for example through clear explanations, scaffolding and feedback — is more important than 
how or when lessons or support are provided. 


To increase access to teaching, it would also be valuable to test the feasibility of online tuition as a way 
to supplement the support provided by teachers to disadvantaged children. 


3. Sustained support will be needed to help disadvantaged pupils catch up 


It is highly likely that the gap will have widened when pupils return to school, even if the strongest 
possible mitigatory steps are put in place. Catch-up provision, including assessment of lost learning 
and targeted support, will be essential. However, it is unlikely that a single catch-up strategy will be 
sufficient to compensate for lost learning due to school closures. 


There is a risk that high levels of absence after schools formally reopen poses a particular risk for 
disadvantaged pupils. 


Limitations 


School closures due to coronavirus are different to the closures included in our rapid evidence 
assessment, meaning that the estimates above should be viewed as an imperfect guide. 


For example, the search provided no examples of unplanned closures of the length already experienced 
by schools in England and the existing evidence on school closures almost exclusively focuses on 
summer holidays and younger children. 


The projections do not incorporate information about activity during school closures due to coronavirus. 
It is possible that factors such as the provision of support for remote learning, or different rates of 
engagement with learning while at home mean that the projections are over- or under-estimates. 


This briefing focuses on learning and does not aim to inform decisions about when pupils return to 
school, which should be based on pupil and teacher safety. 


Introduction 


Background and rationale for the review 


The Covid-19 pandemic has led to school closures across the UK and many countries across the world, 
with the majority of pupils in these systems out of school, though supported and taught in various ways. 
There has been great concern that school closures will lead to slower rates of learning or learning loss, 
and there is a risk that the negative impact will be worse for pupils who are economically disadvantaged. 


In this context a number of researchers and policy organisations have produced quick analyses of the 
potential impact of the school closures (e.g. Sims, 2020; Burgess and Sievertsen, 2020; Kuhfeld, & 
Tarasawa, 2020). These are impressive in their speed and relevance for policy thinking, but they 
highlight the diversity and potentially contested nature of the evidence that may be relevant. Some of 
the earlier reviews of the impact of school closure, although still widely cited, have been subjected to 
considerable criticism (e.g. von Hippel, 2019). 


A rapid evidence assessment seeks to address this heterogeneity by ensuring that, as far as possible, 
all relevant evidence has been captured and considered. We believe the most recent systematic review 
of the evidence on summer learning loss is Cooper et al.’s (1996) study, and have not found any 
systematic review that covers the impact of other causes of closure (e.g. due to epidemics and adverse 
weather), or that focuses specifically on the differential impact of closure on disadvantaged pupils. 


Previous research on the impact of school closures 


The Cooper et al. (1996) meta-analysis has been a key source of evidence about the impact of summer 
closures and has been widely cited. Our inclusion date of 1995 onwards was set partly in order to 
capture any studies not included in that review. Cooper et al. reviewed a history of more than a hundred 
years of research on summer learning loss but focused their meta-analysis on studies published since 
1975, of which they found 13 (two from Canada, all the others from the US; median publication date 
1981). The headline estimate for summer learning loss was 10% of a standard deviation, or about one 
month of learning, slightly higher in maths and lower in reading, and increasing with age, at least in 
reading. They estimated that in reading and language, “on average, summer vacations created a gap 
of about 3 months between middle- and lower-class students” (p261). However, “the meta-analysis 
revealed no differential effect of summer on the mathematics skills of middle- and lower-class students” 
(p261). We should note that summer vacations in the US are typically around three months, about twice 
as long as those in England. 


Despite its dominance in the field of summer learning loss, we believe the Cooper et al. (1996) meta- 
analysis suffers from a number of limitations that reduce the relevance of its claims to our questions. 
Some of these limitations derive from the technical methodological issues we discuss below, for 
example, problems of the scaling and standardisation of test scores. We consider two additional 
concerns here that are more specific to the Cooper et al study: the SES comparison and weightings. 


SES comparison. Our primary focus for this review is the impact of school closure on the disadvantage 
gap: the interaction between the amount of summer learning loss and students’ socioeconomic status 
(SES). Part of the concern here is that the operationalisation of SES in the studies reviewed by Cooper 
et al. (1996) is not very clear. Different study populations were described as 'middle-income' or ‘low- 
income’ without much more detail: students in 28 samples were described as coming from low-income 
families, and students in 20 samples were described as coming from middle-income families. Generally, 
this assessment was based on the community served by a participating school or on the percentage of 
students in a sample who were eligible for free or reduced-price lunch (Cooper et al., p.252). 


A bigger concern is that the comparison of effect sizes for income groups was largely a between-studies 
comparison. This allows considerable scope for confounds: any differences between study populations, 
measures of learning, other variables collected and/or controlled for, or analysis methods could affect 


their estimates. Comparing these estimates across studies mixes any genuine differences in rates of 
learning loss for different SES groups with these different artefacts, many of which are known to be 
capable of affecting the results substantially (von Hippel and Hamrock, 2019). A better approach, and 
the one we have adopted in our analysis, is to draw estimates of the impact on the gap from studies 
that compared the impact for both groups. That way, some of the main study-level artefacts are better 
controlled, since they are likely to affect both groups equally. 


Weightings. Cooper et al. (1996) were faced with something of a no-win dilemma when their systematic 
search process included a single study that was four times as big as all the others combined. The 
Sustaining Effects Study from 1976 (confusingly abbreviated to SES in their paper) had already been 
the subject of a good deal of controversy as different researchers used different analytical approaches, 
different subsets of the data and different interpretations of the same results to argue different positions 
(Cooper et al. devote pages 247-250 to discussion of this history). 


Most problematically, this large study appeared to find no evidence of overall summer learning loss. 
The standard approach in a meta-analysis is to weight the different study estimates, so that those with 
more precision (usually because they are larger) count more. However, in this case, a weighted average 
would simply represent the result from this one study — a positive (i.e. summer learning gain) effect of 
2% of a standard deviation. Instead, Cooper et al. seem to prioritise either the unweighted mean effect 
size, or an estimate with the Sustaining Effects Study removed — with negative effects of 9% and 13% 
of a standard deviation, respectively. Their justification for this is largely that the Sustaining Effects 
Study included a longer interval between tests (140 days) that included about 8 weeks of instructional 
time. However, the average for all studies was 131 days, so it is not clear that this study was an outlier 
in that respect. 


Methodological challenges in evaluating the impact of closure on the gap 


Some of the more recent analyses of summer learning loss draw attention to a range of methodological 
issues and demonstrate that they can make a considerable difference to the estimate of the gap- 
widening effect of closure. We consider three methodological issues here: interval scales, 
standardisation, and analytical choices. 


Interval scales. \f we want to compare the gap between two groups, either on two different tests, or 
even on the same test at different points on the scale, we need to know that the intervals between 
scores on the test are equal across that range. Defining what is meant by ‘equal’ is not simple — we can 
easily get drawn into a complex technical argument about the nature of measurement (e.g. Perline et 
al., 1979) — but an equivalent change in learning must correspond to the same difference in scores. An 
extreme example where this fails would be a test with a ceiling effect, where candidates with quite 
different amounts of learning could be awarded the same (maximum) score. 


Even where there are not clear ceiling (or floor) effects, most tests have different numbers of questions 
targeted at different levels of difficulty and hence differences in the number of marks associated with 
an equivalent change in performance at different points on the scale. Partly for this reason, modern 
tests generally use Item Response Theory (IRT) models to create equal-interval scales instead of just 
scoring as ‘number correct’. Von Hippel and Hamrock (2019) provide a detailed and clear explanation 
of how this problem can lead to the appearance of gap-widening that is a pure artefact of non-interval 
scales. 


Standardisation and reliability. A related problem arises when different tests are used at the two time 
points. Most of the earlier studies, and some of the best known, estimate summer learning gaps by 
testing students at the end of one school year with a test designed for that grade, and then testing again 
at the start of the next with a different test for the next grade. Although this may seem like a necessary 
and perhaps obvious way to proceed, it generates problems for comparing gaps on two quite different 
tests. Without proper vertical scaling using IRT, the usual approach is to standardise the tests; in other 
words, to subtract each group’s mean and divide by its standard deviation. Unfortunately, under this 


procedure, if two tests have different reliabilities, and we split the group into two sub-groups with 
different means, the expected means of the standardised scores for each sub-group will not be equal. 


For example, if the first test is less reliable (which is often the case, for example, as children move from 
Kindergarten to Grade 1), its standard deviation is inflated by random error and hence standardisation 
leads to shrinkage: the mean of a high-SES subgroup is depressed, while the mean of a low-SES 
subgroup is raised. The result is that the gap appears to have widened on the second test, even if 
nothing actually changed. 


According to von Hippel and Hamrock (2019) these two measurement artefacts of interval scales and 
standardisation account for much of what has been claimed as a gap widening effect: 


There are well-known findings suggesting that substantial test score gaps accumulate over 
summer vacation, but those findings were obtained using test scales that spread with age 
and fixed-form tests that change at the end of the summer. Patterns of summer gap growth 
do not necessarily replicate when using modern adaptive tests that are scored on IRT ability 
scales. If summer learning gaps are present, most of them are small and hard to discern 
through the fog of potential measurement artifacts (von Hippel and Hamrock, 2019, p.75). 


Different analytical choices. In any analysis there are choices to be made, some of which affect the 
results. Sometimes these are arbitrary choices where there is not a clear best option, but results will 
nevertheless differ. Sometimes there is a best way, but researchers do not choose it. Sometimes the 
choice reflects a different framing of the question: if you ask a different question you get a different 
answer. An example of this last kind is provided by Quinn et al. (2016) who show that we could think 
about a change in the scores of two subgroups either in terms of their absolute difference on an interval 
scale, or in terms of their relative overlap. Each approach is defensible as answering an important 
question about the gap-widening effects of the school year and summer vacation, but the answers they 
give are not the same. Similarly, Dumont and Ready (2020) frame the choice as an example of Lord’s 
paradox, but also introduce a further dimension of choice: whether the disadvantage gap is defined 
between individual students who differ in their socioeconomic status, or between the students who 
attend schools with differing socioeconomic composition. Again, each of the resulting four choices leads 
to a different conclusion and, according to Dumont and Ready (2020), these differences largely account 
for the different perspectives and conflicting claims among different groups of scholars. 


A further twist is that both Quinn et al. (2016) and Dumont and Ready (2020) analyse the same dataset, 
from the ECLS-K:2011 survey. Our analysis suggests that when they ask the same question, their 
results are pretty close, although not identical. 


Understanding ‘learning loss’ 


The studies we reviewed do not consider the question of what is meant by ‘learning’ and ‘learning loss’ 
in this context. Although a number of studies do compare the effects of school closure on different tests 
measuring different kinds of learning, and some even offer theoretical explanations for these 
differences, there is little consideration of the nature of the learning entailed and whether it is lost or has 
merely become rusty with disuse. 


The distinction has implications for the remedy. If learning has been truly lost, it must be regained, which 
may be slow and painful. On the other hand, if it is merely rusty, it may be quickly regained with a small 
amount of practice. If students have not used a particular technique or procedure for a few months they 
are unlikely to perform it fluently if tested on arrival back in school. But if they had previously learnt it 
well, they might well regain that state quickly. 


In other words, to understand fully the implications of learning loss, we need to know something about 
the process of learning regain. If that process is slow and effortful then the loss is painful. However, if 
the regain is quick and easy then we probably should not even call it ‘loss’. Unfortunately, none of the 
studies we have reviewed tells us anything about the trajectory of learning regain. 


One study that might inform this issue is Kuhfeld and Soland (2020). They find that when test scores 
are available at three points in the year, rates of growth are higher at the beginning of the year and slow 
towards the end. A possible explanation is that part of the growth at the beginning of the year is ‘regain’ 
that is acquired more quickly. Kuhfeld and Soland also show that this departure from linear growth has 
implications for estimates of summer learning loss that may be only half those derived from assuming 
linear growth. 


Methods 
Aims 
The aims for the rapid evidence assessment are captured by the final research questions: 


1. What evidence currently exists about the impact of different kinds of school closure (e.g. due 
to summer holidays; adverse weather, natural disasters)? 


a. On differential academic attainment for disadvantaged/others? 
b. On other outcomes related to education (e.g. impact on IQ or lifetime earnings)? 


2. What factors moderate the impact? (e.g. age of pupils; subject/content area; types of attainment 
measure/methodology; length of closure; timing/conditions of testing)? 


3. What evidence and theory helps us to understand the mechanisms by which school closure 
leads to learning loss and widening of attainment gaps (if it does)? 


In the original design, there were additional research questions about the overall impact of closures on 
learning loss and about the estimate of the absolute effects of schooling. However, these foci were 
dropped at an early stage because of constraints of time. This process is described in the protocol 
which is publicly available on the EEF website. 


Search and screening of studies 


Although the rapid evidence assessment followed an explicit and transparent search process, we do 
not describe it as a systematic review: the process was systematic, but far from comprehensive, given 
constraints of time. Searches were run through Web of Science, ERIC and Google Scholar. Additional 
references were also found through the reference lists of included studies. The exact search terms, and 
inclusion/exclusion criteria can be found in the protocol. The PRISMA flowchart (Moher et al., 2009) for 
inclusion and exclusion of studies at each stage of the process is shown in Figure 1. 
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(Moher et al., 2009) for inclusion of studies 


Screen on 
Title & 
Abstract 


| Ne} AU] B) earoya) 
Title & Abstract 
(193) 


Screen on 
Full Text 
(189) 


INCLUDE on Full 
aa (st:)) 


Yel =1=1 me) a) 
revised 
(oid inslate) 


INCLUDE and 
calculate effect 
sizes (11) 


10 


(010) B) 1 Dela} 
Miiutomcy-Vekiua-leis 
(843) 


EXCLUDED on 
Full Text (131) 


=>,4@ 0) D) De) a) 
revised criteria 
(47) 


The original full-test screening process delivered 58 eligible studies. Given the time and resource 
available, this was more than we would be able to extract data from. At this point, we made a decision 
to focus the review on estimates of the effect of school closure on the gap between disadvantaged 
pupils and others, rather than on estimating the overall effect on learning loss. As many of the remaining 
studies neither attempted to estimate this gap nor provided enough detail for us to be able to extract a 
quantitative estimate, the scale of the review was reduced considerably. 


Data extraction and coding 


Data extraction was conducted on the selected set of 11 studies, in accordance with the protocol. The 
main variables extracted were: 


e Reason for school closure 

e Year in which closure occurred 

e Country 

e Age of pupils affected 

e Number of pupils affected 

e Subject or curriculum area in which learning assessed, and the test used 

e Duration of the closure, and whether the analysis adjusted for imperfect overlap between that 
and the time between tests 

e Whether the sample contained a full range of attainment 

e How the study defined and operationalised the groups being compared (e.g. disadvantaged vs 
not), whether according to individual or group variables 

e The dataset or study used 

e Estimate of delta, the effect size change in the gap in population standard deviation units, per 
month of closure 

e Estimate of the standard error of delta 

e If there were duplicates in the dataset or sample analysed, whether that study provided the best 
estimate available 


Analysis 


Calculating comparable estimates 


For each study-outcome pair we sought to calculate A, defined as the rate of change for the gap 
between “disadvantaged” students and their peers, measured in effect-size units per month of closure.' 


In some instances, this required us to: 


1. Rescale reported effects, so that the units were “effect-size per month”. Equivalent rescaling 
was also applied to estimates of uncertainty. 

2. Rescale reported effects so that they reflected a socioeconomic gap that was as-close-as- 
possible to the “disadvantage gap” in England. This is discussed below. 


The studies in our meta-analysis operationalized disadvantage in different ways. We included studies 
where attainment gaps were defined at the student level. Gaps were defined by: income, poverty (e.g. 
Free or Reduced Price Lunch status), parental occupational, parental education, or some combination 
of these variables. This information is provided for each study in Table 2. 


In England, the educational disadvantage gap at KS2 in 2018-19 is defined by comparing the mean 
attainment rank of the 30.5% of disadvantaged pupils, to the mean attainment rank of the other 59.5% 


wee 


' We followed the definition of “disadvantage” used by the Department for Education, i.e. ““[d]isadvantaged pupils 
are defined as: those who were registered as eligible for free school meals at any point in the last six years, 
children looked after by a local authority or have left local authority care in England and Wales through adoption, 
a special guardianship order, a residence order or a child arrangements order.” 
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of students.? The midpoint of the disadvantaged group is the 15" percentile of the “disadvantage” 
distribution, while the midpoint of the non-disadvantaged group is the 65'" percentile. In terms of 
disadvantage, ‘the gap’ represents 50 percentiles. Assuming a normal distribution, this is equivalent to 
(0.65) — ©(0.15) = 1.45 standard deviation units.° If a research study presented the impact of school 
closures on a disadvantage gap defined as the difference between the mean attainment of the 10" and 
90" percentile (which represents a gap of 2.46 standard deviation units) then we divided the reported 


estimate by ~~, and made equivalent changes to uncertainty estimates. 


Meta-analysis procedure 


Let A; be the i‘ estimate (for i = 1,...,15 ) of the rate at which the gap changes. Assume that these 
estimates have some distribution with mean(A,)=, and var(A;)=t?. At this stage, we make no 
assumptions about the shape of the distribution of A;. We do, however, make a distributional assumption 
about the sampling variance: A;|A,~N(A;, a7). 


We estimate t? using method of moments:* 


—(k-1 
ft? = max < 0, Q-6 - (1) 
iG; = aa? 


Where 
Q=) (4-1) a? @) 
And 
~ LA,67? 
A=sa7 © 


In equation (1), k is the number of independent estimates.® In the interests of conservatism, we limit 
ourselves to setting k equal to the number of unique datasets available for analysis (n=8). This widens 
the predictive interval, described below. 


Following Higgins et al. (2009), we estimate fi as follows: 


YA; 
Lj{O; 


A= (4) 


Where 


wo; = (G7 + #7) (5) 


2 Department for Education data “National curriculum assessments at key stage 2, 2019 (revised)” Available at: 
https://www.gov.uk/government/statistics/national-curriculum-assessments -key-stage-2-2019-revised. 

3 ® represents the normal CDF. 

4 Higgins, J. P., Thompson, S. G., & Spiegelhalter, D. J. (2009). A re-evaluation of random-effects meta-analysis. 
Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1), 137-159. 

5 This could also be estimated from the data, although estimates would be highly uncertain. See Killip, S., 
Mahfoud, Z., & Pearce, K. (2004). What is an intracluster correlation coefficient? Crucial concepts for primary 
care researchers. Annals of Family Medicine, 2(3), 204-208. doi:10.1370/afm.141. 
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Next, we generate empirical Bayes estimates of A for each study-outcome pair: 
a + Gj 
Aj =Ayfit (1 —-A,)A;, where A; = pineee (6) 
0; 


Finally, we generate a distribution of AY°”. This distribution is used in Figure 1 to communicate 
uncertainty about what the impact of a lengthy school closure might be. Again, we are guided here by 
Higgins et al. (2009). Specifically, we make the following distributional assumption:® 


ANew =. fi 
~tx-2 (7) 


2 


| #2 4 (sea) 


Figure 1 contains historical data on the attainment gap between disadvantaged pupils’ and their peers, 
at the end of Key Stage 2. We chose the KS2 measure because primary schools are the overwhelming 
focus of research into school closures and attainment gaps. We selected KS2, rather than Early Years, 
based on our judgement that this was a more reliable measure over time. 


The primary data historical data source in Figure 1 is the Department for Education data “National 
curriculum assessments at key stage 2, 2019 (revised)” (DfE, 2019b). We use table N5. The table 
reports the disadvantage index in which: 


Comparisons are made by ordering pupil scores in reading and maths assessments at 
end of key stage 2 and assessing the difference in the average position of disadvantaged 
pupils and others. The mean rank of pupils in the disadvantaged and other pupils groups 
are subtracted from one another and multiplied up by a factor of 20 to give a value 
between -10 and +10 (where 0 indicates an equal distribution of scores). 


Let D, be the disadvantage index in year t, and p# be the proportion of disadvantaged children in year 
t. Next, let r@ be the mean rank of disadvantaged children (on a scale of 0-100), and r@ be the 
equivalent rank for non-disadvantaged pupils. 


d_,d 
ue _ We use reported values of D, and p, to calculate the 


Based on the above description: D, = 
attainment gap in effect size units. The expression for D, implies: 


ré=5-D,-ré (8) 
The weighted average rank must be 50, implying: 

50 = prt + (1 — pf (9) 
Combining (8) and (9) and rearranging, we have: 


8 A simple alternative would be to use the observed distribution of A. However, this has two shortcomings: first the 
distribution of A is overdispersed (as it contains both t? and a7); second, 7 is estimated with uncertainty, which 
isn’t accounted for in the empirical distribution of A, as per Higgins et al. (2009). 

7 We follow the Department for Education definition: “[d]isadvantaged pupils are defined as: those who were 
registered as eligible for free school meals at any point in the last six years, children looked after by a local 
authority or have left local authority care in England and Wales through adoption, a special guardianship order, a 
residence order or a child arrangements order.” 
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To calculate the attainment gap in effect size units, we assume that attainment follows a normal 
distribution. Using (8) and (10): 


Where @ is the normal CDF, and 6, is the attainment gap defined in terms of effect size. 


Moderator analysis 


Our systematic review only found 15 comparable estimates of A;, making it difficult to perform useful 
moderator analysis. In particular, existing literature contained limited variation in outcomes (which were 
almost all “reading” and “maths”) and age. 


That said, we tested whether there were any clear differences in A; for subject and age. 
First, we fit a simple linear model to see if reading gaps seemed to grow faster than those maths: 
A; = Bo + BR°*4Read; +e; (model 1) 


Where Read; is a binary indicator equal to one if the outcome of A; was reading. We fit model 1 to the 
13 estimates of maths/reading, using inverse-variance weights. The point estimate was BRe¢4 = 
—0.012. We then conducted a simple randomization inference. The null hypothesis being examined 
was £*e2¢ = 0. The test statistic was the t-statistic associated with B®°*¢from model 1. The observed 
test statistic was t°S= -1.17. To generate a single draw under the null, we randomized the “subject” 
variable, re-fit model 1 and captured the t-statistic. We repeated this process 10000 times. The p-value 
can be defined as the proportion of draws under the null with an absolute value greater than the ¢°°s.8 
In this case, p=0.332. In short, we find no evidence of an association between outcome-type and the 
rate at which gaps. Given our power to detect such an association, this comes as no surprise. 


We conducted a similar procedure in terms of age. This time we fit model 1 to our full meta-analytic 
sample of 15 estimates: 


A; = Bo + B49° Age; +e; (model 2) 


Age is defined by the average age of the children reported in the study. Again, this model was fit with 
inverse-variance weights. There was no evidence of association between Age; and dj. 


8 Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University Press. 
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Results 


The extracted effect size estimates and other relevant variables from all the studies that provided 
quantitative estimates are shown in Table 1. 


Table 1 

Studies where we could calculate FSM6 gap 
Study_year Cohort Country Grade_range _n_student Subject _ Break length Dataset Delta_gap SE_Delta_gap 
burkham_2004 1999 USA K-1 3664 Other 2.6 ecls_k99 0.049 0.011 
davies_2013 2011 Canada 1-3 1376 Reading 2.2 davies 0.011 0.004 
dumont_2020/quinn_2016 2012 USA K-1 3740 Maths 2.6 ecls_k11 0.043 0.006 
dumont_2020/quinn_2016 2012 USA K-1 3750 Reading 2.6 ecls_k11 0.016 0.006 
dumont_2020/quinn_2016 2013 USA 1-2 3630 Maths 2.6 ecls_k11 -0.001 0.005 
dumont_2020/quinn_2016 2013 USA 1-2 3630 Reading 2.6 ecls_k11 -0.021 0.005 
lindahl_2001 1998 Sweden 5-6 556 Maths 2.2 indahl 0.009 0.024 
meyer_2017 2013 Germany 2-3 51 Reading 1.4 meyer 0.113 0.076 
meyer_2017 2013 Germany 2-3 51 Other 1.4 meyer -0.019 0.057 
paechter_2015 2013 Austria 5-6 180 Maths 2.1 paechter 0.073 0.013 
verachtert_2009 2003 Belgium K-1 829 Maths 2.0 verachtert 0.012 0.029 
vonhippel_2019 1999 USA K-1 17779 Maths 2.6 ecls_k99 0.014 0.005 
vonhippel_2019 1999 USA K-1 17779 Reading 2.6 ecls_k99 0.015 0.004 
vonhippel_2019 1986 USA 1-6 790 Maths 2.6 bss 0.047 0.012 
vonhippel_2019 1986 USA 1-6 790 Reading 2.6 bss 0.033 0.014 

Studies with other measures of disadvantage (not comparable) 
Study_year Cohort Country Grade_range _n_student Subject _ Break length _ Dataset Delta_gap SE_Delta_gap 
campbell_2019 2016 USA 3-4 5513 Reading 2.6 campbell -0.027 0.007 
meyer_2017 2013 Germany 2-3 78 Reading 1.4 meyer 0.163 0.079 
meyer_2017 2013 Germany 2-3 78 Other 1.4 meyer 0.054 0.065 
meyer_2020 2014 NZ 4-7 4390 Other 1.4 nz_govt 0.166 0.029 
vonhippel_2019 1999 USA K-1 17779 Maths 2.6 ecls_k99 0.014 0.007 
vonhippel_2019 2009 USA K-1 177549 Maths 2.6 nwea -0.005 0.005 
vonhippel_2019 1999 USA K-1 17779 Reading 2.6 ecls_k99 0.026 0.007 
vonhippel_2019 2009 USA K-1 177549 Reading 2.6 nwea -0.019 0.005 
vonhippel_2019 2009 USA 1-6 177549 Maths 2.6 nwea -0.011 0.002 
vonhippel_2019 1986 USA 1-6 790 Maths 2.6 bss 0.034 0.012 
vonhippel_2019 2009 USA 1-6 177549 Reading 2.6 nwea -0.007 0.002 
vonhippel_2019 1986 USA 1-6 790 Reading 2.6 bss 0.057 0.014 
vonhippel_2019 2009 USA 1-8 177549 Reading 2.6 nwea -0.014 0.002 
vonhippel_2019 2009 USA 1-8 177549 Maths 2.6 nwea -0.011 0.002 


Table 1 contains information on the following variables: 


Study_year: first author, and the year in which the study was published 

Cohort: year in which data was collected 

Country 

Grade range: reported grade range of students 

n_students: number of students reported in the study 

Subject: “Other” represents either the general knowledge test in the ECLS-K study, or writing 
tests 

Break length: how long were schools closed? (measured in months) 

Dataset: indication about the underlying dataset used. bss = Beginning School Study; ecls = 
early childhood longitudinal study; nwea = GRD study, maintained by the northwest evaluation 
association. 

Delta_gap [A]: main outcome variable defined as the rate of change in the gap between FSM6 
and nonFSM6 pupils, measured in effect-size units per month 

SE_Delta_gap: (SE(A)): standard error of A 


The set of studies that appear in Table 1 are: 


All studies we use in our meta-analysis. This includes all studies where we have been able to 
calculate an estimate of A. In cases where multiple authors analysed the same sample of 
children, we have removed the studies with clear methodological deficiencies. For one dataset, 
ecls_k11, we found two analyses of excellent quality (Dumont 2020 and Quinn 2016). In order 
to avoid double counting, we took an average of the estimates from these analyses. 

All other studies where we found quantitative estimates of how SES gaps changed during 
school closures, but where disadvantage was measured at the school level, or in some way 
that could not be credibly converted into measure comparable with the FSM gap. 
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Table 2 partially repeats information from Table 1, but is limited to the nine studies that provided the 15 
estimates that are comparable and of high quality, and provides additional information about the 
definition of disadvantage used in each study. It also shows the Bayesian shrunken estimates, A;, that 
represent our best estimate of the likely contribution of each study to an overall measure. These results 
are also shown graphically in a forest plot, in Figure 2. 


Table 2 
x a a 

Disadvantage Aj SE(A;) (shrunken 
Study_year Country n_student Subject Definition (delta_gap) (standard error) estimates) 
burkam_2004 USA 3664 Other SES! 0.049 0.011 0.043 
davies_2013 Canada 1376 Reading SES? 0.011 0.004 0.011 
dumont_2020/quinn_2016 USA 3630 Reading SES? -0.021 0.005 -0.018 
dumont_2020/quinn_2016 USA 3630 Maths SES? -0.001 0.005 0.000 
dumont_2020/quinn_2016 USA 3750 Reading SES? 0.016 0.006 0.017 
dumont_2020/quinn_2016 USA 3740 Maths SES? 0.043 0.006 0.041 
lindahl_2001 Sweden 556 Maths SES* 0.009 0.024 0.017 
meyer_2017 Germany 52 Other Parental occupation® _—-0.019 0.057 0.018 
meyer_2017 Germany 51 Reading Parental occupation® 0.113 0.076 0.028 
paechter_2015 Austria 182 Maths Mother’seducation® 0.073 0.013 0.059 
verachtert_2009 Belgium 829 Maths SES’ 0.012 0.029 0.019 
vonhippel_2019 USA 17779 Maths FRPL status® 0.014 0.005 0.014 
vonhippel_2019 USA 17779 Reading FRPL status’ 0.015 0.004 0.015 
vonhippel_2019 USA 790 Reading FRPL status® 0.033 0.014 0.030 
vonhippel_2019 USA 790 Maths FRPL status® 0.047 0.012 0.041 


1Composite measure of parents' education, parents’ occupational prestige, and household income; composite measure of parent 
education, other parent education, income each standardized and summed; 7NCES-created socioeconomic status (SES) variable, which is 
a composite of family income, parental education, and occupational prestige; ‘Census-based measure, combining the mean income and 
mean parental years education on the block of the relevant student, among households on that block where parents are aged 28-54 and 
kids are aged 10-12; Highest Socio-Economic Index of Occupational Status; ° Binary indicator for whether or not a mother sat the 
university entrance exam; ‘Composite measure including the educational level of both parents, the professional status of both parents, and 
the household income; ’Free and Reduced Price Lunch status. 
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Figure 2: Change in attainment gap (by study and outcome) 
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Figure 2 presents the overall estimate from the meta-analysis of 0.022 (standard error = 0.006) standard 
deviations per month, as the amount by which the attainment gap between disadvantaged pupils and 
their peers may be expected to grow. 


Figure 3 presents this estimate in the context of the existing and historical gap in attainment at Key 
Stage 2. The thick blue line is the median projection and covers the second half of the 2019-20 
academic year. The uncertainty in our projection is illustrated by the two light blue lines. These represent 
the 25th and 75th percentile and can be thought of as plausible "good" and "bad" cases. The median 
estimate, based on the existing literature on school closures, indicates that the gap would widen by 
36% by September 2020. The range displayed stems from 11% to 75%. The projections suggest that 
school closures will widen the attainment gap between disadvantaged children and their peers, likely 
reversing progress made to narrow the gap since 2011. 


It should be acknowledged that this range of estimates only contains 50% of our uncertainty. In other 
words, if we were to observe a new estimate of A”°” (for example from a new study that we had not 
previously seen) we would expect the effect size to fall outside of the 25-75" percentile range half of 
the time. In addition, it should be noted that some studies contained estimates indicating that in certain 
subjects and for some age groups the gap may narrow. 
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Figure 3: historical estimates of the FSM gap at KS2 
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Wider literature and limitations 


It is important to stress that the current school closures due to coronavirus are different to the 
closures included in our systematic review, meaning that the estimates above should be viewed as an 
imperfect guide. For example, our search provided no examples of unplanned closures of the length 
already experienced by schools in England and the existing evidence on school closures almost 
exclusively focuses on summer holidays and younger children. 


One clear difference between closures due to coronavirus and the closures analysed above is that 
schools have been providing substantial support to enable remote learning, including by providing 
resources and online teaching. In addition, national initiatives such as Oak Academy have been 
watched by millions of pupils (Schools Week, 2020). 


A recent review of remote teaching conducted by the EEF found that remote learning can be effective, 
given the right conditions (EEF, 2020a). A review of the impact of online schools in the US found that 
although pupil outcomes online were, on average, poorer for all groups compared to in-school 
learning, attainment gaps between disadvantaged students and their peers were no wider 
(Woodworth, 2015). This might suggest that, if online schooling were operating well for all children, 
then the projection of gap widening above may be overstated. 


It is also likely that some pupils will return to school earlier than September. For these pupils, the 
projection of gap widening in Figure 3 may also be overstated. 


However, while efforts to support remote learning are likely to have been of considerable benefit to 
many children, and are likely to have reduced the overall amount of learning loss due to closures, 
there are indications that, overall, the remote learning that has taken place during school closures is 
likely to have further widened rather than narrowed the gap. 
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Surveys of teachers and parents in England in 2020 show that many pupils are not engaging in high- 
quality home learning and that disadvantaged pupils appear to be learning less than their peers (e.g. 
Sutton Trust, 2020; Institute for Fiscal Studies, 2020). Findings from these studies suggest that 
children from the most disadvantaged families are spending less time on learning activities, are 
submitting less work and typically have access to fewer resources at home. 


There are several additional reasons why the studies reviewed here may underestimate the impact of 
school closures. For example, the estimates do not capture the fact that coronavirus has had a 
differentially large economic and health impact on disadvantaged families (e.g. Office for National 
Statistics, 2020; Douglas et al., 2020), which may in turn affect educational outcomes. 


Given the lack of evidence about how coronavirus-specific factors might affect the rate at which the 
attainment gap will widen, we have not attempted to include these factors in our quantitative 
projections. 


Estimates by subject, phase and prior attainment 


Testing for differences across subject and age is severely limited by two factors. First, there is a lack 
of variation in existing research. All but two of the studies focus on either reading or maths, and all the 
estimates in our meta-analysis come from research conducted in primary schools. Second, we have a 
small set of estimates to draw on, as our systematic review only yielded 15 comparable estimates. 


With those important caveats in mind, we note that we did not find any evidence of an association 
between gap estimates and age or subject. It is worth stressing that these findings are an “absence of 
evidence”, rather than clear evidence of no difference. The EEF hopes to address this shortage of 
evidence in future work. 


Similarly, there is little evidence related to differences between pupils with high or low prior 
attainment. Gershenson (2017) finds that over the summer higher-attaining disadvantaged children 
fall behind other higher attainers at a faster rate than other groups. However, this result was not 
replicated in maths, and overall there was not enough evidence to draw clear conclusions. 
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Discussion and implications 


Given the impact on the gap identified by the rapid evidence assessment, we now explore evidence to 
inform efforts to mitigate the extent to which the gap widens and to compensate for lost learning, 
including by drawing on wider literature on effective approaches (e.g. EEF, 2020a). 


Supporting learning at home 


Two factors affecting learning while pupils are at home are remote learning and parental involvement. 
Remote learning 


It is very hard to use technology to replace the learning relationships that exist between teachers and 
pupils in the classroom. However, providing access to teaching via technology has the potential to 
make a small-to-moderate positive impact on learning during school closures. 


A key challenge is ensuring that access to teaching is provided to all pupils. There is a significant risk 
that disadvantaged children have less access to teaching than their peers, in part due to having 
reduced access to technology, exacerbating the impact of school closures on the attainment gap. 


A rapid evidence assessment on remote learning conducted by the EEF (available here) also 
emphasised that the pedagogical quality of remote learning is more important than how lessons are 
delivered. Ensuring the elements of effective teaching are present — for example; clear explanations, 
scaffolding and feedback — is more important than how or when they are provided (EEF, 2020a). It is 
unlikely that providing pupils with access to resources without support will improve learning. 


To increase access to teaching, it would also be valuable to test the feasibility of online tuition as a 
way to supplement the support provided by teachers to disadvantaged children. 


In addition to providing access to technology, ensuring that teachers and pupils are provided with 
support and guidance to use specific platforms is essential, particularly if new forms of technology are 
being implemented (EEF, 2020a). 


Parental involvement 


Parental engagement in children’s learning and the quality of the home learning environment are 
associated with improved academic outcomes at all ages (EEF, 2020b). 


However, the evidence indicates that it is very challenging for schools to increase levels of parental 
engagement successfully. Schools may need support in communicating effectively with parents and in 
helping parents understand specific ways to help their child learn. 


It is likely to be particularly valuable to focus on developing and maintaining two-way communication 
with parents and promoting the development and maintenance of reading habits. 


The effectiveness of strategies will differ by age group. For example, in primary schooling, shared 
book reading and linked activities such as building vocabulary and practising spellings are valuable, 
while in secondary schools parents can support children to read independently and create study 
routines (e.g. Meyer et al., 2015). 


Parents can support their children by encouraging them to set goals, plan, and manage their time, 
effort, and emotions. This type of support can help children, in particular older children, to regulate 
their own learning and will often be more valuable than direct help with schoolwork. 


EEF resources for schools on supporting parents during school closures are available here. 
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Supporting catch-up after pupils return to school 


It is highly likely that the gap will have widened when pupils return to school, even if the strongest 
possible mitigatory steps are put in place. Approaches that could help pupils catch up include: 


e Targeted support 
e Professional development for teachers 


Key risks related to pupil absence and sustained support are also highlighted. 
Targeted support 


The EEF has identified a list of 18 promising projects that have been evaluated and shown to have 
positive impacts on learning, with particularly strong effects for disadvantaged children in most cases. 


Tuition is likely to be a particularly effective catch up approach. The EEF estimates that the average 
impact of one-to-one tuition is five additional months’ progress (EEF, 2020b). An evaluation of low- 
cost tutoring delivered by university students showed a positive impact on learning of three additional 
months’ progress (Torgerson, 2018). 


Professional development 


Alongside targeted interventions, improving the quality of teaching is the strongest lever schools have 
to improve pupil outcomes, particularly for disadvantaged students. 


The EEF recommends that when spending the pupil premium schools take a tiered approach, starting 
with efforts to improve teaching quality. 


Priorities for professional development might include: ensuring high-quality materials are available for 
early career teachers linked to the Early Career Framework; online courses linked to the best 
available evidence on improve literacy and maths; and online courses linked to pedagogical 
approaches that are likely to be particularly effective for disadvantaged learners, e.g. metacognition. 


Pupil absence 


A key risk relates to the distinction between school closures and pupil absence. Notwithstanding the 
overall projections above, Goodman (2014) emphasises that schools are typically able to deal 
relatively effectively with school closures, be they planned or unplanned. In contrast, missing school 
due to absence is typically associated with a substantially greater negative effect. 


Part of this difference is likely to be driven by methodological challenges, i.e. there are likely to be 
unobservable factors that are associated with being absent that lead to low attainment, even when 
pupils compared to apparently similar peers. However, it is also likely that it is easier for teachers to 
respond to closures — for example, by repeating key content as a class — than it is to support 
individual children who have been absent (e.g. Department for Education, 2016; Gottfried, 2010). 


The severe negative effect of absence poses a particularly high risk for disadvantaged children 
(Department for Education, 2019b), who typically have lower rates of attendance and whose families 
have indicated that the they would be substantially less likely to send their child back to school if given 
the choice (IFS, 2020). 


Sustained support 
Sustained support will be required to help disadvantaged pupils catch-up after they return to school. 
While a focused catch-up programme - including assessment and targeted support — would be 


beneficial when pupils first return to school, it is unlikely that a single catch-up strategy will be 
sufficient to compensate for lost learning due to school closures. 
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Additional resources 


The Education Endowment Foundation has created a number of resources that are relevant to 
supporting learners during the Covid-19 outbreak. All resources can be found here. Some of the 
resources that directly relate to the findings of this rapid evidence assessment are detailed below: 


Resource Description Link 
Guidance EEF guidance reports provide clear and | https://educationendowmentfoundation.org.uk/tools/guidan 
reports actionable recommendations for ce-reports/ 

teachers on a range of high-priority 

issues based on the best available 

evidence. 
Parental Four recommendations on working with | httos://educationendowmentfoundation.org.uk/tools/quidan 
engagement parents to support their child’s learning. | ¢ce-reports/working-with-parents-to-support-childrens- 


guidance report 


learning/ 


Parental 


The underlying evidence review for the 


https ://educationendowmentfoundation.org.uk/evidence- 


engagement parental engagement guidance report. summaries/evidence-reviews/parental-engagement/ 
evidence review 

Digital Four recommendations on using digital https://educationendowmentfoundation.org.uk/tools/guidan 
technology technology to improve children’s ce-reports/using-digital-technology-to-improve-learning/ 
guidance report | learning. 

Digital The underlying evidence review for the https://educationendowmentfoundation.org.uk/evidence- 
technology digital technology guidance report. summaries/evidence-reviews/digital-technology-2019/ 


evidence review 


Metacognition 
guidance report 


Seven recommendations for teaching 
self-regulated learning and 
metacognition, 


https://educationendowmentfoundation.org.uk/tools/quidan 
ce-reports/metacognition-and-self-requlated-learning/ 


Teaching and 
Learning 
Toolkit 


The Teaching and Learning Toolkit 
provides an accessible summary of the 
evidence across 35 different 
approaches aimed at improving pupil 
outcomes 


https ://educationendowmentfoundation.org.uk/evidence- 
summaries/teaching-learning-toolkit/ 


Peer tutoring 


Toolkit summary of peer tutoring 
approaches 


https://educationendowmentfoundation.org.uk/evidence- 
summaries/teaching-learning-toolkit/peer-tutoring/ 


Metacognition 


Toolkit summary of metacognition and 


https ://educationendowmentfoundation.org.uk/evidence- 


and self- self-regulation approaches summaries/teaching-learning-toolkit/meta-cognition-and- 
regulation self-requlation/ 
Parental Toolkit summary of parental https://educationendowmentfoundation.org.uk/evidence- 
engagement engagement approaches summaries/teaching-learning-toolkit/parental-engagement/ 
Homework Toolkit summary of homework (primary https://educationendowmentfoundation.org.uk/evidence- 
and secondary) summaries/teaching-learning-toolkit/homework-primary/ 
https ://educationendowmentfoundation.org.uk/evidence- 
summaries/teaching-learning-toolkit/nhomework-secondary/ 
Digital Toolkit summary of digital technology httos://educationendowmentfoundation.org.uk/evidence- 
technology approaches summaries/teaching-learning-toolkit/digital-technology/ 
EEF-funded This is the full list of evaluations that https://educationendowmentfoundation.org.uk/projects- 
evaluations have been funded by the EEF. and-evaluation/projects/ 
What Works A list of studies that examine the impact | https://ies.ed.gov/ncee/wwe/distancelearningstudy 


Clearinghouse 
list of studies 


of remote learning approaches, 
identified by the What Works 
Clearinghouse 
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